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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re Patent Application of 
Ullas Gargi et al. 
Application No.: 09/904,627 
Filed: July 16, 2001 



For: HIERARCHICAL IMAGE FEATURE- 
BASED VISUALIZATION 



Group Art Unit: 2672 
Examiner: JIN CHENG WANG 
Confirmation No.: 2970 



RECEIVED 

APR 3 0 2004 

Technology Center 2600 



REQUEST FOR RECONSIDERATION 



N Commissioner for Patents 
P.O. Box 1450 
Alexandria, VA 22313-1450 

Sir: 

In response to the Office Action mailed January 28, 2004, reconsideration and 
allowance of the present application are respectfully requested. 
Claims 1-20 remain pending in the application. 

On page 2 of the Office Action, claims 1-7, 11-12 and 16-19 are rejected 

under 35 U.S.C. §1 02(e) as being anticipated by U.S. Patent No. 6,240,423 (Hirata 

et al). The Examiner asserts in numbered paragraph 3 with respect to claim 1 : 

Hirata teaches a method of visualizing and retrieving a data file 
comprising: 

Displaying a plurality of images representing corresponding data files 
on a display device using a first distance metric (images being 
retrieved in order based upon the distance between the query image 
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and the result candidate images, column 6) between each data file 
(e.g., figures 1-11; column 6-9); 

Redisplaying a portion of the images on the display device using a 
refined distance metric (e.g., figures. 9, 11 and 12; column 6-9); and 

Performing at least one of retrieving, marking and selecting at least one 
desired data file (Hirata teaches retrieving at least one desired data file; 
see, figures 1-11; column 6-9). 



Later, on pages 5-6 of the Office Action, in numbered paragraph 4, the 
Examiner rejects independent claim 17 based on the Hirata patent. In this portion of 
the Office Action, the Examiner asserts that the Hirata patent discloses clustering 
image data files using a distance metric in Figures 1-11 and at column 6-9. The 
Examiner also asserts that the Hirata patent discloses: 



Interactively selecting, by a user, a portion of the images (e.g., a user 
specify [ies] the regions of the images; e.g., the region division of figure 
3; figures 1-11; column 6-9); 

Redisplaying the portion of the images in real time on the display 
device using a redefined distance metric (e.g., re-order based on the 
similarity among the candidate images of figure 9 or 2 nd stage image 
matching based on boundary of figure 11 and grouping and re-ordering 
based on the similarity among candidates using distance metric of 
figure 11; figures 1-11; column 4); and 

Retrieving a desired data file (retrieved data file is displayed on a 
display device; figures 1-11; column 4). 

On pages 7-9 of the Office Action, claims 8 and 13-14 are rejected as being 

unpatentable over the Hirata patent in further view of U.S. Patent No. 6,584,221 

(Moghaddam). On pages 9-10 of the Office Action, claims 9-10 and 20 are rejected 

as being unpatentable over the Hirata patent in view of U.S. Patent No. 6,121 ,969 
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(Jain). On pages 10-1 1 of the Office Action, claim 15 is rejected as being 
unpatentable over the Hirata patent in view of U.S. Patent No. 5,528,259 (Bates). 

The foregoing rejections are respectfully traversed, as the Hirata patent, 
considered alone, or in combination with the various secondary references relied 
upon by the Examiner, fails to teach or suggest all features set forth in Applicants' 
independent claims 1 and 17. 

The present application discloses a multi-media database and 
classification system, which can, for example, provide for automatic classification 
and retrieval of multimedia files based on features of the multimedia files. Figure 1 
shows an exemplary method for visualizing and retrieving data files in accordance 
with an exemplary embodiment of the present invention. In step 1 10, a plurality of 
images representing data files on a display device are displayed using a first (i.e., 
coarse) distance metric. The distance metric represents a distance between each 
data file. In step 120, a portion of the images can be redisplayed on the display 
device using a refined distance metric. As described on specification page 6 
beginning with line 4, the distance metric can be refined at each of plural redisplays 
until a desired data file is found or a maximum refined distance metric is reached. In 
step 140, a desired data file can be retrieved, and/or can be marked or selected. 
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Figures 2A-2E show a graphical representation of displays generated in 
accordance with exemplary embodiments, where each portion of the images 
redisplayed can be graphically selected by a user. Figure 2A is a screen capture of 
an exemplary display showing a first level of images representing data files on a two- 
dimensional display. A first, coarse, distance metric can be calculated, and allows 
the user to receive useful information about the organization of images on the 
display. Based on perceived properties of a desired image, a user can select an 
area 202 of the screen where a desired image most likely resides. Figure 2B shows 
a portion of images redisplayed as selected by area 202. Here the distance metric 
has been recalculated using more of the image information than was used in the first 
distance calculation. As described on specification page 9, the reclustering and 
redisplaying of selected images is more than merely a zoom function. 

The redisplayed process can be repeated, as represented by Figures 2C and 
2D. Distance metrics between images are again recalculated using more image 
feature data than previously used with respect to Figure 2B. Following the redisplay 
based on recalculated distance metrics, a desired data file (e.g., image 240) is 
identifiable and can be selected and retrieved as shown in Figure 2E. Figure 3 
shows another exemplary method which can be implemented in accordance with the 
present invention, and Figure 4 shows a flowchart for performing a coarse to fine 
distance calculation. 

The foregoing features are broadly encompassed by independent claims 1 
and 17. Exemplary embodiments of the present invention display images using a first 
distance metric between each data file, and provide for redisplaying a portion of the 
images using a refined distance metric in a manner which constitutes more than 
merely a zoom feature. Claim 1 is directed to a method of visualizing and retrieving a 
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data file from a second data file. The claim 1 method includes a step of displaying a 
plurality of images representing corresponding data files on a display device using a 
first distance metric between each data file. Claim 1 also recites redisplaying a 
portion of the images on the display device using a refined distance method. Claim 
17 is directed to a method of interactively retrieving a data file from a set of data files 
in real time, and includes features similar to those mentioned with respect to claim 1 . 

The foregoing features are neither taught nor suggested by the Hirata patent 
considered individually or in combination with the various secondary documents 
relied upon by the Examiner. 

The Hirata patent is directed to a method for querying a database of images 
wherein the results of the query are based upon a combination of region based 
image matching and boundary based image matching. In Hirata's query-based 
system, an initial query is submitted and a set of results is clustered. The clustering 
of the search results is performed using a distance between each file and a distance 
to the query, as discussed at column 6, lines 37-41. In all cases, clustering is based 
on the distance to a query (see Hirata's Figures 8 and 9). Figure 1 1 of the Hirata 
patent discloses a second stage of image matching; however, this second stage of 
image matching, like the first stage, is applied to all images initially obtained in 
response to a query. 

In contrast to the method disclosed and recited in Applicants' independent 
claims 1 and 17, the Hirata patent fails to teach or suggest using a refined distance 
metric which is applied to a user selected subset of displayed images. 

In addition, the Hirata patent fails to teach or suggest the claim 17 feature of 
interactive user selection for selecting a subset of images to be displayed. Rather, 
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the query based system of the Hirata patent groups all images, and only creates new 
groupings based on a new query. The Hirata patent does not teach or suggest 
applying a second stage of image matching to a user selected subset of returned 
images. 

In rejecting claim 17 on page 6 of the Office Action, the Examiner refers to the 
region division in figure 3 of the Hirata patent as illustrating user specified regions. 
However, Hirata does not teach or suggest that a refined distance metric, as 
presently claimed, is used to redisplay the images of any defined region (see Hirata 
at column 4, line 10 to column 5, line 13). As such, the Hirata patent also fails to 
teach or suggest the user interactive selection feature of Applicants claim 17. 

Although Hirata uses a first stage of region based matching, and a second 
stage of boundary based matching, both of these image display processes operate 
on the initial data set acquired in response to the initial query. Hirata does not teach 
or suggest redisplaying a portion of the initial images using a refined distance metric. 

Thus, the Hirata patent fails to teach or suggest the features recited in 
Applicants' independent claims 1 and 17. 

The remaining documents relied upon by the Examiner; namely, the patents 
to Moghaddam, Jain and Bates, were discussed in Applicants' previous response, 
and for reasons similar to those discussed therein, fail to overcome the deficiencies 
of the Hirata patent. As such, independent claims 1 and 17 are considered 
allowable. All of the other claims depend from independent claims 1 and 17 and 
recite additional advantageous features which further distinguish over the documents 
relied upon by the Examiner. 
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All rejections and objections raised in the Office Action having been 
addressed, it is respectfully submitted that the present application is in condition for 
allowance and a Notice of Allowance is respectfully solicited. 

Respectfully submitted, 

Burns, Doane, Swecker& Mathis, l.l.p. 





Date: April 28, 2004 By: 

Patrick C. l Keane 
Registration No. 32,858 

P.O. Box 1404 

Alexandria, Virginia 22313-1404 
(703) 836-6620 



